Annotating a presentation in a telepresence meeting

ABSTRACT

A processing system can include an encoder to encode a real-time transmission of a presentation. A memory buffer can copy and store images of the presentation and convert the images into snapshot images. A transmitter can transmit the snapshot images to an external annotation device, and a receiver can receive annotation data of an annotation performed on the snapshot images at the external annotation device. The annotation can be encoded, in accordance with the annotation data, into the real-time transmission of the presentation to display the real-time transmission with the annotation.

TECHNICAL FIELD

This application relates generally to annotating a presentation in atelepresence, video conference or interactive meeting.

BACKGROUND

In typical telepresence and video conferencing meetings, users can sharepresentations by sending them as encoded video streams.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete appreciation of the disclosed embodiments and many ofthe attendant advantages thereof will be readily obtained as the samebecomes better understood by reference to the following detaileddescription when considered in connection with the accompanyingdrawings, wherein:

FIGS. 1-3 schematically illustrate exemplary hardware architectures andthe use thereof in exemplary presentation scenarios with annotation;

FIGS. 4A-4C illustrate exemplary algorithms for annotating procedures;and

FIG. 5 illustrates an exemplary processing system.

DESCRIPTION OF EXAMPLE EMBODIMENTS OVERVIEW

According to exemplary implementations, a presentation in a telepresencemeeting can be annotated, and the annotation can be executed by one ormore processors, circuits, and computer hardware components, which canbe configured to execute corresponding processes by software programmingor discrete circuit designs, and can be distributed across a number ofdifferent machines.

A processing system can include an encoder to encode a real-timetransmission of a presentation. A memory buffer can copy and storeimages of the presentation and convert the images into snapshot images.A transmitter can transmit the snapshot images to an external annotationdevice, and a receiver can receive annotation data of an annotationperformed on the snapshot images at the external annotation device. Theannotation can be encoded, in accordance with the annotation data, intothe real-time transmission of the presentation to display the real-timetransmission with the annotation. The annotation data can also be sentto another processing system, such as a processing system that isconnected to the originating processing system by the Internet or othernetwork, where the annotation data is encoded into the real-timetransmission by the another processing system.

The memory buffer can store raw images of the presentation in an uncodedstate and first and second copies of the raw images. The encoder canencode the raw images, the first copy of the raw images or the secondcopy of the raw images from the memory buffer to form the real-timetransmission. A selection of which of the images to select can be basedon the status of displaying an annotation. The snapshot images can beformed from the first copy of the raw images or second copy of the rawimages in the memory buffer.

The annotation can be drawn in the memory buffer according to theannotation data in real-time on the second copy of the raw images in thememory buffer. Annotated snapshot images can be formed from the secondcopy of the raw images in the memory buffer and transmitted to theexternal annotation device, to provide real-time feedback of drawing theannotation on the external annotation device. The first copy of the rawimages in the memory buffer can be utilized for forming the snapshotswhen the annotation is cleared. When the annotation is being coded intothe real-time transmission of the presentation, the encoder can utilizethe second copy of the raw images as annotated raw images.

The memory buffer can include logically or physically separated first,second and third buffers. The first buffer can store the raw images ofthe presentation in the uncoded state, and the second and third bufferscan respectively store the first and second copies of the raw images.When the annotation is drawn in the memory buffer according to theannotation data in real-time on the second copy of the raw images in thememory buffer, the third buffer can store annotated raw images, which isa copy of the raw images stored in the first buffer having theannotation drawn thereon, and the second buffer can store un-annotatedraw images, which is a copy of the raw images stored in the first bufferhaving no annotation drawn thereon.

The memory buffer can be a single buffer that stores raw images of thepresentation in an uncoded state as well as copies (e.g., first andsecond copies) of the raw images. The memory buffer can also includelogically or physically separated first, second and third buffers. Thefirst buffer can store raw images of the presentation in an uncodedstate, and the second and third buffers can each store a copy of the rawimages from the first buffer.

The encoder can encode the raw images from the first buffer to form thereal-time transmission, and the snapshot images can be formed from thecopy of the raw images in the third buffer. The memory buffer can drawthe annotation according to the annotation data in real-time on the copyof the raw images in the third buffer, such that annotated snapshotimages are formed from the raw images in the third buffer andtransmitted to the external annotation device, to provide real-timefeedback of drawing the annotation on the external annotation device.The copy of the raw images in the second buffer can be utilized forforming the snapshots when the annotation is cleared. The annotation canbe encoded into the real-time transmission of the presentation by theencoder utilizing annotated raw images in the third buffer.

The annotation data can be scaled to account for differences in sizeratios between a display of the external annotation device and a displayformat of the real-time transmission. In some aspects, the annotationdata can be scaled to account for pixel ratio differences between thedisplay of the external annotation device and the display format of thereal-time transmission. For example, the display of the externalannotation device can be a 4:3 screen (e.g., 640×480, 1024×768 or2,048×1,536 pixels) and the display format of the real-time transmissioncan be 16:9 (e.g., 640×360, 854×480, 960×540, 1024×576, 1280×720,1366×768, 1600×900, 1920×1080, 2048×1152, 2560×1440, 3840×2160 or4096×2304 pixels). However, other screen formats can be utilized ineither the external annotation device or the real-time transmission,including wide-screen or cinema formats, 16:10, 5:3 and others.

The real-time transmission of the presentation can have a transmissionvideo format that remains the same after the annotation is coded intothe real-time transmission. In some aspects, a remote processing systemthat receives can receive a real-time transmission that has the sameformat and decoding requirements regardless of whether an annotation isincluded in the real-time transmission.

A display to display the presentation in a raw format can be provided. Afield programmable gate array can be provided to route the presentationbetween the memory buffer and the display, and to accept thepresentation as an input from a computer. A network adapter thatincludes the transmitter and the receiver can be provided.

The presentation can include one or more of a document, a whiteboard, avideo, a still image and text.

The receiver can receive annotation data of annotations performed on thesnapshot images at a plurality of external annotation devices, includingthe external annotation device and other external annotation devices.The annotations can be drawn, in real-time, on the images stored in thememory buffer to form annotated images. The snapshot images can beformed from the annotated images, and transmitted to the externalannotation devices to provide real-time feedback of the annotations ateach of the external annotation devices. The encoder can encode thereal-time transmission, in real-time, based on the annotated images sothat the real-time transmission provides a real-time of the annotations,as the annotations are being input to the external annotation devices.The system can further include the external annotation devices.

A processing system can include a decoder to decode a real-timetransmission of a presentation. A memory buffer can copy images of thepresentation and convert the images into snapshot images. A transmittercan transmit the snapshot images to a local external annotation devicethat is located locally to the processing system, and a receiver canreceive local annotation data of an annotation performed on the snapshotimages at the local annotation device. The local annotation data can betransmitted to a source of the real-time transmission to be encoded intothe real-time transmission, so that the real-time transmission includesthe annotation.

The real-time transmission of the presentation can include an annotationfrom a remote annotation device that is located remotely to theprocessing system. The source of the real-time transmission can beanother processing system that is connected to the processing system bya network adapter and via the Internet.

Multiple sources of annotation data are possible, in which a processingsystem can, in real-time, plot or draw corresponding annotations on apresentation and encode the annotations and the presentation into areal-time transmission. The multiple sources of annotation data can belocal to the processing system, remote to the processing system (andconnected thereto by a network such as the Internet), or a combinationof local and remote to the processing system. Remote multiple sources ofannotation data can each be located at separate remote processingsystems, and more than one remote source of annotation data can belocated at a single remote processing system.

A method can include encoding a real-time transmission of apresentation; copying and storing images of the presentation, andconverting the images into snapshot images; transmitting the snapshotimages to an external annotation device; and receiving annotation dataof an annotation performed on the snapshot images at the externalannotation device, where the annotation is encoded, by a processor inaccordance with the annotation data, into the real-time transmission ofthe presentation to display the real-time transmission with theannotation. A non-transitory computer-readable medium includingcomputer-executable instructions that, when executed by computerprocessor, can cause the computer processor to execute the method. Stepsof the method or a corresponding computer process can be executed byvarious means, including encoding, converting, decoding, processing,transmitting, and receiving means that correspond to computer hardwarecomponents or computer hardware components that are configured bycomputer software or computer code.

DETAILED DESCRIPTION OF EXEMPLARY IMPLEMENTATIONS

In the drawings, like reference numerals designate identical orcorresponding parts throughout the several views. Further, as usedherein, the words “a,” “an” and the like generally carry a meaning of“one or more,” unless stated otherwise.

Aspects of this disclosure include providing an annotation on apresentation (e.g., on snapshots of the presentation) and incorporatingthe annotation into existing presentation streams in telepresencesystems. An exemplary telepresence system is a BFCP (binary floorcontrol protocol) or H.239-compatible telepresence/UC (unifiedcommunications) endpoint.

According to aspects of this disclosure that are discussed below inrelation to exemplary implementations, a procedure for performing anannotation can include:

A. Taking a snapshot (SS) of an image (local or remote), and storing acopy of the SS in a codec (i.e., a main processing unit).

B. Sending another copy of the SS to a touch/annotation device.

C. Annotating: a user draws/inputs annotations on the touch/annotationdevice (for local users), and the device sends annotation data (e.g.,annotation vectors) to the codec.

D. Encoding and transmitting an image (e.g., a real-time transmission)that is encoded with annotations as continuous video to remote and/orlocal participants.

In the exemplary procedure discussed above, annotations can be updatedlive or in real-time continuously as the annotation is being created atthe external annotation device (e.g., an in-touch device), so that aremote participant or other user can view that annotation as it is beingcreated by a different user. Annotations can be made on any presentationshared by remote participants irrespective of a capability of anendpoint that shares the presentation. The annotation can be providedand updated in real-time, i.e., remote users can see the annotations(e.g., drawings) as they are being made and with very little delay.Remote users on endpoints that have no annotation capability are able tosee the presentation and annotation naturally. In other words, if alocal user (i.e., a user that is local to the processing unit that isgenerating/encoding a presentation) is drawing a circle, then the remoteuser should see the circle as it is being drawn, rather than merelyseeing a circle after it has been fully drawn.

Annotations can be shown on the annotation device used to generate theannotation as well as on data displays available at the local endpoint(EP). Similar to remote users, the presentation and annotation should beseen naturally on the annotation device, and with even less delay thanthe case of remote users. Annotations can also be created, at the sametime, by various users, in real-time, regardless of whether the usersare local, remote, or a combination thereof.

Exemplary steps in exemplary implementations include: taking apresentation snapshot, whether it is shared locally or remotely; sendingthe snapshot to the control input device for displaying, where thesnapshot is ready for annotation; and blending the annotation on theinput device and at the local display.

Exemplary effects provided by the exemplary implementations includeproviding a blending ability of any screen size, with any type ofbackground graphic format with annotation metadata, while sending out anannotated video as a standard video format that can be universallyreceived by a device that is capable of receiving the standard videoformat, and a simple tool to scale, plot and blend a drawings (anannotation) in a way that the drawings appears as if it is drawn inreal-time at a remote endpoint. The simple tool can handle multiplesources of metadata (annotation vectors), and blend it in real-time.

According to aspects of this disclosure, a legacy EP that has acapability to view standard video streams can view the annotationdrawings in real-time (i.e., can display or view the circle drawing atthe remote in real-time, instead of displaying the circle at the remoteafter circle is complete drawn), and multiple sources, of differentgraphic size, can be blended together in real-time. For instance, aplurality of users connected to each other through a plurality ofdifferent EPs can each be assigned a different annotation marker orcolor, so that separate annotations can be easily discerned, andcorresponding coordinates (e.g., vectors) of annotations from differentusers can be simultaneously plotted and blended, and then encoded. Whilethe legacy EP can view annotations, as discussed herein, the legacy EPmay not be able to accept and/or act as a transceiver for metadatarelating to an annotation or annotation data, such that the legacy EPcannot act as a source of annotation data. Consequently, in-touch orother display devices connected to the legacy EP may not be able toperform annotations in accordance with the disclosure, absent softwareor hardware modifications to further the capabilities of the legacy EPto include required functionality and/or hardwareconfigurations/architecture.

In some exemplary implementations, a content channel can be treated as avideo stream. In particular, data is captured, modified (withannotations), encoded, and transmitted as video frames. Frames can beencoded using a conventional or other video coding method.

From a user's perspective, aspects of this disclosure can be consideredas treating content as a video stream. A video stream from any source,e.g., a camera pointing to a document, whiteboard, etc., may be capturedand annotated. Combined with a triple-buffer architecture, a view can bereturned to a non-annotated view (or switched in-between easily) withouthaving to interrupt a real-time encoding or decoding process. Further,annotations can be shared in real-time to a first group of participants(such as, e.g., only local users), and then the annotations can beshared in real-time to a second group of participants (such as, e.g.,remote users) only have the local users initiate annotation encoding.The local/remote users division is exemplary. The first group ofparticipants can include a combination of local and remote users, andthe second group of participants can include a combination of otherlocal and remote users. The first group of participants can also includea combination of local and remote users, and the second group ofparticipants can include a combination of other remote users. Anycombination can be defined prior to a teleconference session or inreal-time.

A triple-buffer implementation allows for a scratch version to be stored(e.g., in a loopback buffer) along with an original snapshot. As aresult, it is possible to show the annotation (the scratch versionstored in the loopback buffer) to local participants (or other usergroup) only, and at any point go back to the original snapshot. In oneexemplary scenario, local participants can take a snapshot, doannotations locally (displayed locally in real-time, but not shown toremote participants) until one point, when the local participants decideto share the annotated version, then the remote participants can takeover, interact, and/or collaboratively annotate together.

In an implementation of decoupled annotation input and display, a systemcan have a display (such as, e.g., a large central screen) that isdecoupled from an annotation screen (which is a screen the annotation isinputted to). Local users can be provided with touch-based tablets, andannotation inputs from the local users on the tablets can be displayedon the decoupled display. However, because of a triple-bufferimplementation, the displays of the tablets and/or the decoupled displaycan each show different images—an annotated image or an original image.Selection of either image can be made by an end-user or a master user orterminal. Further, the “original” image can be replaced with adifferently annotated image. Thus, in a triple-buffer implementation twodifferent sets of annotations can be stored for display.

FIG. 1 schematically illustrates an exemplary hardware architecture andthe use of the hardware in an exemplary scenario when the presentationis locally shared.

In FIG. 1, a personal or portable computer (PC) provides a presentationto an FPGA (field programmable gate array) that accepts thepresentation, and feeds the presentation to a DSP (digital signalprocessor). In FIG. 1, the DSP is in an encoding mode, which causes theDSP to encode, in the DSP Encoder, raw images (video) of thepresentation into an RTP (real-time protocol or secure real-timeprotocol) stream/transmission, which is output to all of the EPs(endpoints) in the meeting. A local display, which is connected to theFPGA, can display the presentation directly from the PC.

The presentation (in raw images, prior to encoding) is buffered in a RawBuffer. The Raw Buffer provides an output to the DSP Encoder and otherbuffers. When an annotation is made on the presentation, a snapshot (SS)of the raw buffer is taken. The SS is copied into two buffers: aSnapshot Buffer and a Loopback Buffer. Further buffers can also beprovided. According to the exemplary implementations described herein,the Snapshot Buffer is utilized for supporting a clearing annotationfeature, which allows the annotation to be cleared.

The SS is also copied and converted to a JPEG (JPG) format, and sent toin In-Touch Device for later displaying on a screen (Touch Screen) ofthe device, so that a user can annotate via the screen. The SS may alsobe copied and converted to another picture file format, other than JPEG,and the SS may also be a frame of a video stream that is copied from theDSP, and sent to the In-Touch Device. Since the In-Touch Device can havea different screen resolution than that of the presentation, or as thatof the local display, a screen resolution conversion can take place inconverting the SS (prior to sending the SS to the In-Touch Device).

In FIG. 1, the main unit that includes the FPGA and the DSP is a Codec,which refers to a main processing machine/device/apparatus. The Codeccan include computer hardware components, which are discussed in furtherdetail below, such as a transmitter Tx (for transmitting screenshots andother information to an In-Touch Device, wirelessly or by wire), and areceiver Rx (for receiving annotation data or other commands from theIn-Touch Device, wirelessly or by wire).

The Codec can include an Annotation Software Component, which is anapplication or software configuration that can cause circuits of aCodec, such as a conventional Codec, to provide special functionalityrelating to annotation. Further, the In-Touch Device can include aUser-Interface (UI) Component, which is an application or softwareconfiguration that can cause circuits of a computing system, such as aconventional tablet or touch-screen-enabled computing device, to providespecial functionality relating to providing a user-interface forannotation. In one exemplary implementation, the In-Touch Device can bea tablet computer, such as one operating a proprietary operating system,such as Microsoft Windows, Android OS, Linux OS, Mac OS, iOS or UNIX,where the UI component can be provided as an installable or installedapplication program that operates on the operating system.

In an exemplary implementation, when a user of the In-Touch Device viewsa SS on the Touch Screen of the In-Touch Device, an annotation can bemade via the UI Component. In particular, coordinates of the annotationcan be recorded by the UI Component, and transmitted to the AnnotationSoftware Component of the Codec. The coordinates can be stored asvectors, which can then be plotted or drawn onto the presentation, inreal-time. Preset or pre-defined shapes can also beinput/stored/transmitted, as well as text, images, video or other data,which can be stored as annotation data that can be sent, in real-time,to the Codec for processing or to the Codec for re-transmission toanother Codec for processing.

In some aspects, a user of the In-Touch Device does not view annotationsdirectly on the Touch Screen in response to performing an annotation.Rather, the annotation is displayed on the Touch Screen in response toannotation data being sent to the Codec, which is then plotted on thepresentation (e.g., on a following SS), and then transmitted back to theIn-Touch Device for display. This is only one exemplary implementation.

In another exemplary implementation, an instantaneous input on the TouchScreen can be shown as a first annotation of a first blending ratio orcolor on the Touch Screen in an unchanged/non-annotated screenshot, viaprocessing performed entirely in the In-Touch Device. Concurrently, theIn-Touch Device can output corresponding annotation data to the Codec.Then, a second annotation of a second blending ratio or color can beprovided by the Codec, in response to receiving the annotation data,with annotated screenshots. The In-Touch Device can set the firstannotation to fade away or disappear after a predefined time, and canthereby provide a more fluid and natural annotating process for a user.The predefined time can be set as a time that is approximately the sameas a calculated delay between the In-Touch Device and the Codec, withrespect to a time required to transmit annotation data, plot theannotation data, and receive/display a corresponding SS on the In-TouchDevice. Also the first and second blending ratios or colors can bedistinct so that the user is aware of any discrepancies. However, theratios or colors can also be the same between the first and secondannotations.

The Raw Buffer, the Snapshot Buffer, and the Loopback Buffer can bediscrete memories, logically discrete portions of a common memorybuffer, or a combination thereof. The Raw Buffer, the Snapshot Buffer,and the Loopback Buffer can be embodied in a single memory buffer.Additional buffers can also be provided for additional annotations thatcan be processed, controlled and displayed separately or independently.

FIG. 2 schematically illustrates an exemplary hardware architecture andthe use of the hardware in an exemplary scenario when the presentationis received from a remote EP. The hardware illustrated in FIG. 2 can bethe same as or substantially similar to that illustrated in FIG. 1.

In FIG. 2, the presentation RTP stream is received (from a remote EP)and is decoded, and fed into the Raw Buffer. A raw frame in the RawBuffer is sent through the FPGA and to the display on the data display.The FPGA can also output the raw frame to the PC, and the RTP or the rawframe can also be recorded on a mass storage device for later retrieval.

Here, it should be appreciated that FIG. 2 can be a conventional Codecthat does not include annotation functions, where the annotations arealready present in the RTP stream (having been encoded therein by, e.g.,the hardware and process described with respect to FIG. 1). However, asillustrated in FIG. 2, the illustrated hardware does include theannotation functions.

When an annotation is made on the presentation, a snapshot of the rawbuffer is taken and copied into two buffers, the Loopback Buffer andSnapshot Buffer. The Snapshot Buffer can provide support for a clearingannotation feature. The raw snapshot is copied and converted to apicture file format, such as a JPG format, and sent to the In-touchdevice for later posting on the screen for a user to annotate.

FIG. 2 schematically illustrates an exemplary hardware architecture andthe use of the hardware in an exemplary scenario of switching to astream from the Loopback buffer. The sequence can proceed by determiningwhether the DSP is loaded with an encoder. If not, the DSP can bereloaded as an encoder. The DSP can be switched from being a decoder oran encoder depending on a direction of a stream.

Once the DSP is reloaded as an encoder, streaming from the Loopbackbuffer is started, and routing from the raw buffer to the FPGA to thedisplay is switched to the Loopback Buffer. The SS is also posted on thein-touch screen for the user to start making annotation.

As the user performs annotating on the SS on the Touch Screen,annotation data (e.g., coordinate data) is sent to the Codec and plottedinto the Loopback buffer in the DSP. Since the DSP encoder continuouslyencodes the raw data from the Loopback Buffer, and sends out the resultto remote end users (remote endpoints), the annotation will be displayedat the remote endpoint naturally, in a manner just similar to how thelocal user annotating his snapshot is displayed the annotation on theIn-Touch device.

In an exemplary implementation, the In-Touch Device can support a1366×768 screen resolution. If a SS is a 1920×1080 resolution, then theSS can be scaled to show in any size on the In-Touch Screen (such as asize that is of more or less resolution than the SS). A size of anannotation input can also be tracked. For instance, a width of theannotation can be tracked by a size of a pen input or a finger input.This width size can be varied by a software scaling. Otherwise, bothcoordinate and the pen size can be used to proportionally scale anannotation before sending the annotation to the codec to plot in theLoopback Buffer. Alternatively, if the Codec has a screen sizeconfiguration of the In-Touch Device stored beforehand, scaling can beperformed by the Codec upon receipt thereof.

An exemplary scaling formula defines the following relationships:Xp=(Wp*Xi)/WiYp=(Hp*Yi)/HiPp=Pi*(Wp*Hp)/(Wi*Hi)

(Xi, Yi)—the drawing coordinate on the scaled image shown on theIn-Touch

(Xp, Yp)—the drawing coordinate on the original snapshot

(Wi, Hi)—the width and height of the picture shown on the In-Touch

(Wp, Hp)—the width and height of the original snapshot

Pi—the drawing pen size on the scaled image shown on the In-Touch

Pp—the drawing pen size on original scaled image

Both coordinate (Xp, Yp) and pen size Pp are sent from the In-Touch tothe codec to plot in the Loopback Buffer. However, as discussed above,the Codec may already have this or corresponding information based on apredefined configuration. The pen size can be adjusted by a user, orbased on a detected input at a device.

By scaling being performed between the Codec and the In-Touch, inputscan be properly translated into annotations, in their intended location,across a plurality of different resolution combinations.

FIGS. 4A-4C illustrate exemplary algorithms for annotating, by way offlow charts, that correspond to the processing and hardware describedabove with respect to FIGS. 1-3 and the exemplary processing systemdescribed in FIG. 5.

In FIG. 4A, a processing system (such as a Codec) encodes and transmitsa presentation at S402. At S404, a snapshot (SS) is taken, and anannotation procedure 406 is performed. At S408, the annotation (e.g.,the annotation data) is plotted onto the presentation, so that theannotation can be displayed on the presentation.

In FIG. 4B, a processing system (such as a Codec) receives and decodes apresentation at S412. At S414, a snapshot (SS) is taken, and anannotation procedure 416 is performed. At S418, the annotation (e.g.,the annotation data) is plotted onto the presentation, so that theannotation can be displayed on the presentation. Since the processingsystem in FIG. 4B may not be an originator of the presentation, theannotation data can be transmitted to a processing system thatoriginated the presentation to plot the annotation data. Also, if thepresentation (i.e., a file of the presentation, such as a movie file ora Microsoft PowerPoint file) is shared across the processing system, aparticular processing system that is forming the RTP can be switchedbased on a source of annotation, e.g., in a situation where a primaryannotator is not local to the original processing system that forms theRTP.

An exemplary annotation procedure consistent with 406 and 416 isillustrated in the flowchart of FIG. 4C. Here, a user input is receivedat S420 by, e.g., a pen and a touch screen. Coordinate Data (e.g.,annotation data) is stored at S422 that corresponds to a coordinatesystem of the touch screen and inputs received by the touch screen. Thedata can be stored as points, vectors, shapes, text or a combinationthereof. Other visible or invisible annotations are also possible. TheCoordinate Data is scaled at S424 (either by the input device or by areceiving device) to account for resolution or dimensional differencesbetween the touch screen, another display screen or a native size of apresentation upon which the annotation is being performed. The scaleddata is then sent to, e.g., a Codec or other Endpoint at S426 forincorporation/plotting on the presentation. The incorporation/plottingcan include blending an annotation with a presentation, such that theannotation is opaque or translucent, allowing a portion of thepresentation to show through a portion of the annotation. Other blendingtechniques can also be applied.

Aspects of the exemplary implementations provide for annotation on apresentation video in real-time. A blending ability of any size and anytype of background in any graphic format, such as jpeg, raw, png, etc.,with annotation data is provided, and a resulting annotated presentationstream can be sent out as a standard video format. In one exemplaryimplementation, the background can be just a clean-slate whiteboardbackground.

Scaling for plotting and blending is provided in such a way that thedrawing will be seen in real-time at the remote endpoints. Annotationdata can be handled local or remote, which allows two or more persons ona same Endpoint to draw simultaneously. The blending function frommultiple annotation sources can be done in real-time, simultaneously, asit appears to two or more persons standing up at a whiteboard drawing atthe same time.

In accordance with the above exemplary algorithms, a process caninclude: taking the presentation snapshot, whether it is shared locallyor remotely, and storing the snapshot stored in a local loopbackannotation buffer in the DSP, were this local loopback buffer iscontinuously encoded to send out as a RTP stream to conference meeting;sending the snapshot to the in-touch for displaying; and, for everyannotation move on the In-Touch Device, corresponding annotation data issent to the Codec and plotted on the local loopback annotation buffer inthe DSP.

Adverting back to FIGS. 1-3, the blocks of architecture shown thereincan be implemented in a single processing system, or distributed acrossa plurality of processing systems, which may be referred to asprocessors, Codecs, Endpoints or other machines and apparatuses. Forinstance, each of the blocks of architecture can be a discreteprocessor, circuit, system or logic component.

FIG. 5 illustrates an exemplary processing system or processor 500. Oneor more of such processing systems can be utilized in or to execute oneor more algorithms, or portions thereof, or one or more architectureblocks, or portions thereof, in accordance with the descriptionsprovided herein.

The exemplary processing system 500 can be implemented using one or moremicroprocessors or the equivalent, such as a central processing unit(CPU) and/or at least one application specific processor ASP (notshown). The microprocessor is a circuit that utilizes a computerreadable storage medium, such as a memory circuit (e.g., ROM, EPROM,EEPROM, flash memory, static memory, DRAM, SDRAM, and theirequivalents), configured to control the microprocessor to perform and/orcontrol the processes and systems of this disclosure. Other storagemediums can be controlled via a controller, such as a disk controller,which can controls a hard disk drive or optical disk drive.

The microprocessor or aspects thereof, in alternate implementations, caninclude or exclusively include a logic device for augmenting or fullyimplementing this disclosure. Such a logic device includes, but is notlimited to, an application-specific integrated circuit (ASIC), a fieldprogrammable gate array (FPGA), a generic-array of logic (GAL), andtheir equivalents. The microprocessor can be a separate device or asingle processing mechanism. Further, this disclosure can benefit fromparallel processing capabilities of a multi-cored CPU. One or moreprocessors in a multi-processing arrangement may also be employed toexecute sequences of instructions contained in memory. Alternatively,hard-wired circuitry may be used in place of or in combination withsoftware instructions. Thus, the exemplary implementations discussedherein are not limited to any specific combination of hardware circuitryand software.

In another aspect, results of processing in accordance with thisdisclosure can be displayed via a display controller to a monitor. Thedisplay controller preferably includes at least one graphic processingunit, which can be provided by a plurality of graphics processing cores,for improved computational efficiency. Additionally, an I/O(input/output) interface is provided for inputting signals and/or datafrom microphones, speakers, cameras, a mouse, a keyboard, a touch-baseddisplay or pad interface, etc., which can be connected to the I/Ointerface as a peripheral. For example, a keyboard or a pointing devicefor controlling parameters of the various processes or algorithms ofthis disclosure can be connected to the I/O interface to provideadditional functionality and configuration options, or control displaycharacteristics. Moreover, the monitor can be provided with atouch-sensitive interface for providing a command/instruction interface.

The above-noted components can be coupled to a network, such as theInternet or a local intranet, via a network interface for thetransmission or reception of data, including controllable parameters. Acentral BUS is provided to connect the above hardware componentstogether and provides at least one path for digital communication therebetween.

Suitable software, such as an operating system or an application, can betangibly stored on a computer readable medium of a processing system,including the memory and storage devices. Other examples of computerreadable media are compact discs, hard disks, floppy disks, tape,magneto-optical disks, PROMs (EPROM, EEPROM, flash EPROM), DRAM, SRAM,SDRAM, or any other magnetic medium, compact discs (e.g., CD-ROM), orany other medium from which a computer can read. The software mayinclude, but is not limited to, device drivers, operating systems,development tools, applications software, and/or a graphical userinterface.

Computer code elements on the above-noted medium may be anyinterpretable or executable code mechanism, including but not limited toscripts, interpretable programs, dynamic link libraries (DLLs), Javaclasses and complete executable programs. Moreover, parts of theprocessing of aspects of this disclosure may be distributed for betterperformance, reliability and/or cost.

The procedures and routines described herein can be embodied as asystem, method or computer program product, and can be executed via oneor more dedicated circuits or programmed processors. Accordingly, thedescriptions provided herein may take the form of exclusively hardware,exclusively software executed on hardware (including firmware, residentsoftware, micro-code, etc.), or through a combination of dedicatedhardware components and general processors that are configured byspecific algorithms and process codes. Hardware components are referredto as a “circuit,” “module,” “unit,” “device,” or “system.” Executablecode that is executed by hardware is embodied on a tangible memorydevice, such as a computer program product. Examples include CDs, DVDs,flash drives, hard disk units, ROMs, RAMs and other memory devices.

Reference has been made to flowchart illustrations and block diagrams ofmethods, systems and computer program products according toimplementations of this disclosure. Aspects thereof are implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in acomputer-readable medium that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablemedium produce an article of manufacture including instruction meanswhich implement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer orother programmable data processing apparatus to cause a series ofoperational steps to be performed on the computer or other programmableapparatus to produce a computer implemented process such that theinstructions which execute on the computer or other programmableapparatus provide processes for implementing the functions/actsspecified in the flowchart and/or block diagram block or blocks.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made without departingfrom the spirit and scope of this disclosure. For example, preferableresults may be achieved if the steps of the disclosed techniques wereperformed in a different sequence, if components in the disclosedsystems were combined in a different manner, or if the components werereplaced or supplemented by other components. The functions, processesand algorithms described herein may be performed in hardware or softwareexecuted by hardware, including computer processors and/or programmablecircuits configured to execute program code and/or computer instructionsto execute the functions, processes and algorithms described herein.Additionally, some implementations may be performed on modules orhardware not identical to those described. Accordingly, otherimplementations are within the scope that may be claimed.

The invention claimed is:
 1. A processing system, comprising: an encoderto encode a real-time transmission of a presentation; a memory buffer tocopy and store images of the presentation and convert the images intosnapshot images including a first copy of the images and a second copyof the images; a transmitter to transmit the snapshot images to anexternal annotation device; and a receiver to receive annotation data ofan annotation performed on the snapshot images at the externalannotation device, wherein the annotation is encoded, in accordance withthe annotation data, into the real-time transmission of the presentationto display the real-time transmission with the annotation, wherein theannotation is drawn in the memory buffer according to the annotationdata in real-time on the second copy of the images in the memory buffer,such that annotated snapshot images are formed from the second copy ofthe images in the memory buffer and transmitted to the externalannotation device, to provide real-time feedback of drawing theannotation on the external annotation device; and wherein the first copyof the images in the memory buffer is utilized for forming the snapshotswhen the annotation is cleared.
 2. The processing system according toclaim 1, wherein: the memory buffer stores the images as raw images ofthe presentation in an uncoded state and first and second copies of theraw images; the encoder encodes the raw images or one of the first andsecond copies of the raw images from the memory buffer to form thereal-time transmission; and the snapshot images are formed from one ofthe first and second copies of the raw images in the memory buffer. 3.The processing system according to claim 1, wherein, when the annotationis being coded into the real-time transmission of the presentation, theencoder utilizes the second copy of the raw images as annotated rawimages.
 4. The processing system according to claim 3, wherein: thememory buffer includes logically or physically separated first, secondand third buffers; the first buffer stores the raw images of thepresentation in the uncoded state; and the second and third buffersrespectively store the first and second copies of the raw images.
 5. Theprocessing system according to claim 4, wherein, when the annotation isdrawn in the memory buffer according to the annotation data in real-timeon the second copy of the raw images in the memory buffer, the thirdbuffer stores annotated raw images, which is a copy of the raw imagesstored in the first buffer having the annotation drawn thereon, and thesecond buffer stores un-annotated raw images, which is a copy of the rawimages stored in the first buffer having no annotation drawn thereon. 6.The processing system according to claim 3, wherein the annotation datais scaled to account for differences in size ratios between a display ofthe external annotation device and a display format of the real-timetransmission.
 7. The processing system according to claim 3, wherein thereal-time transmission of the presentation has a transmission videoformat that remains the same after the annotation is coded into thereal-time transmission.
 8. The processing system according to claim 1,further comprising: a display to display the presentation in a rawformat.
 9. The processing system according to claim 8, furthercomprising: a field programmable gate array to route the presentationbetween the memory buffer and the display, and to accept thepresentation as an input from a computer.
 10. The processing systemaccording to claim 1, further comprising: a network adapter thatincludes the transmitter and the receiver.
 11. The processing systemaccording to claim 1, wherein the presentation includes one or more of adocument, a whiteboard, a video, a still image and text.
 12. Theprocessing system according to claim 1, wherein: the receiver is toreceive annotation data of annotations performed on the snapshot imagesat a plurality of external annotation devices, including the externalannotation device and other external annotation devices; the annotationsare drawn, in real-time, on the images stored in the memory buffer toform annotated images; and the snapshot images are formed from theannotated images, and transmitted to the external annotation devices toprovide real-time feedback of the annotations at each of the externalannotation devices.
 13. The processing system according to claim 12,wherein the encoder encodes the real-time transmission, in real-time,based on the annotated images so that the real-time transmissionprovides a real-time of the annotations, as the annotations are beinginput to the external annotation devices.
 14. The processing systemaccording to claim 13, further comprising the external annotationdevices.
 15. A processing system, comprising: a decoder to decode areal-time transmission of a presentation; a memory buffer to copy imagesof the presentation and convert the images into snapshot imagesincluding a first copy of the image and a second copy images; atransmitter to transmit the snapshot images to a local externalannotation device that is located locally to the processing system; anda receiver to receive local annotation data of an annotation performedon the snapshot images at the local annotation device, wherein the localannotation data is transmitted to a source of the real-time transmissionto be encoded into the real-time transmission, so that the real-timetransmission includes the annotation, wherein the annotation is drawn inthe memory buffer according to the annotation data in real-time on thesecond copy of the images in the memory buffer, such that annotatedsnapshot images are formed from the second copy of the images in thememory buffer and transmitted to the local external annotation device,to provide real-time feedback of drawing the annotation on the localexternal annotation device; and wherein the first copy of the images inthe memory buffer is utilized for forming the snapshots when theannotation is cleared.
 16. The processing system according to claim 15,wherein the real-time transmission of the presentation includes anannotation from a remote annotation device that is located remotely tothe processing system.
 17. The processing system according to claim 16,wherein the source of the real-time transmission is another processingsystem that is connected to the processing system by a network adapter.18. A method, comprising: encoding a real-time transmission of apresentation; copying and storing images of the presentation, andconverting the images into snapshot images including a first copy of theimages and a second copy of the images; transmitting the snapshot imagesto an external annotation device; and receiving annotation data of anannotation performed on the snapshot images at the external annotationdevice, wherein the annotation is encoded, by a processor in accordancewith the annotation data, into the real-time transmission of thepresentation to display the real-time transmission with the annotation,wherein the annotation is drawn according to the annotation data inreal-time on the second copy of the images, such that annotated snapshotimages are formed from the second copy of the images and transmitted tothe external annotation device, to provide real-time feedback of drawingthe annotation on the external annotation device; and wherein the firstcopy of the images is utilized for forming the snapshots when theannotation is cleared.